{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelling the risk of default for credit card applicants\n",
    "\n",
    "This program evaluates the risk of default for credit card loan applicants. It uses logistic regression to determine model coefficients and then feeds those coefficients into a binary classification model. The binary classification model has also been built in Excel and can be viewed here: https://goo.gl/PNTDd8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from patsy import dmatrices\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import data for both training and testing, and optionally (if known) input the **cost per false positive** and the **cost per false negative**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "training_raw = pd.read_csv('training.csv', low_memory=False)\n",
    "test_raw = pd.read_csv('test.csv', low_memory=False)\n",
    "\n",
    "cost_per_false_positive = 2500\n",
    "cost_per_false_negative = 5000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ease of modelling later, add a column to the data which is simply the reverse of the default outcome:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_raw['outcomes_rev'] = (training_raw.outcomes != 1).astype('int')\n",
    "test_raw['outcomes_rev'] = (test_raw.outcomes != 1).astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add a column which is the ratio of total debt to income for each application, and also a column which is the square of the given columns to see if a more complex model can provide more accurate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_raw['debt_to_income'] = (training_raw['credit_card_debt'] + \n",
    "                              training_raw['automobile_debt']) / training_raw['income']\n",
    "test_raw['debt_to_income'] = (test_raw['credit_card_debt'] + \n",
    "                              test_raw['automobile_debt']) / test_raw['income']\n",
    "\n",
    "training_square = pd.DataFrame()\n",
    "test_square = pd.DataFrame()\n",
    "\n",
    "cols_to_square = ['age','years_employment','years_at_address','income',\n",
    "                  'credit_card_debt','automobile_debt','debt_to_income']\n",
    "\n",
    "training_square[cols_to_square] = training_raw[cols_to_square].apply(lambda x: x**2)\n",
    "test_square[cols_to_square] = test_raw[cols_to_square].apply(lambda x: x**2)\n",
    "\n",
    "training_square.rename(columns={'age':'age2','years_employment':'years_employment2',\n",
    "                                'years_at_address':'years_at_address2','income':'income2',\n",
    "                                'credit_card_debt':'credit_card_debt2',\n",
    "                                'automobile_debt':'automobile_debt2',\n",
    "                                'debt_to_income':'debt_to_income2'}, inplace=True)\n",
    "test_square.rename(columns={'age':'age2','years_employment':'years_employment2',\n",
    "                            'years_at_address':'years_at_address2','income':'income2',\n",
    "                            'credit_card_debt':'credit_card_debt2',\n",
    "                            'automobile_debt':'automobile_debt2',\n",
    "                            'debt_to_income':'debt_to_income2'}, inplace=True)\n",
    "training_raw = pd.concat([training_raw, training_square], axis=1)\n",
    "test_raw = pd.concat([test_raw, test_square], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalizing the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_norm = pd.DataFrame()\n",
    "test_norm = pd.DataFrame()\n",
    "\n",
    "cols_to_norm = ['age','years_employment','years_at_address','income',\n",
    "                'credit_card_debt','automobile_debt', 'debt_to_income', \n",
    "                'age2','years_employment2','years_at_address2','income2',\n",
    "                'credit_card_debt2','automobile_debt2', 'debt_to_income2']\n",
    "training_norm[cols_to_norm] = training_raw[cols_to_norm].apply(\n",
    "        lambda x: (x - x.min()) / (x.max() - x.min()))\n",
    "test_norm[cols_to_norm] = test_raw[cols_to_norm].apply(\n",
    "        lambda x: (x - x.min()) / (x.max() - x.min()))\n",
    "\n",
    "training_norm.rename(columns={'age':'age_norm','years_employment':'years_employment_norm',\n",
    "                'years_at_address':'years_at_address_norm','income':'income_norm',\n",
    "                'credit_card_debt':'credit_card_debt_norm',\n",
    "                'automobile_debt':'automobile_debt_norm',\n",
    "                'debt_to_income':'debt_to_income_norm', \n",
    "                'age2':'age2_norm','years_employment2':'years_employment2_norm',\n",
    "                'years_at_address2':'years_at_address2_norm','income2':'income2_norm',\n",
    "                'credit_card_debt2':'credit_card_debt2_norm',\n",
    "                'automobile_debt2':'automobile_debt2_norm',\n",
    "                'debt_to_income2':'debt_to_income2_norm'}, inplace=True)\n",
    "test_norm.rename(columns={'age':'age_norm','years_employment':'years_employment_norm',\n",
    "                'years_at_address':'years_at_address_norm','income':'income_norm',\n",
    "                'credit_card_debt':'credit_card_debt_norm',\n",
    "                'automobile_debt':'automobile_debt_norm',\n",
    "                'debt_to_income':'debt_to_income_norm', \n",
    "                'age2':'age2_norm','years_employment2':'years_employment2_norm',\n",
    "                'years_at_address2':'years_at_address2_norm','income2':'income2_norm',\n",
    "                'credit_card_debt2':'credit_card_debt2_norm',\n",
    "                'automobile_debt2':'automobile_debt2_norm',\n",
    "                'debt_to_income2':'debt_to_income2_norm'}, inplace=True)\n",
    "\n",
    "training_raw = pd.concat([training_raw, training_norm], axis=1)\n",
    "test_raw = pd.concat([test_raw, test_norm], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and set initial intercepts to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_raw['intercept'] = 1\n",
    "test_raw['intercept'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to define several helper functions to run the model.\n",
    "\n",
    "First up, is a function to calculate the thresholds for the binary classification model. It simply finds the maximum and minimum scores and makes a number of evenly spaced out bins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_thresholds(df, num_thresholds):\n",
    "    '''\n",
    "    helper function to calculate the threshold values\n",
    "    '''\n",
    "    threshold_step = (df['scores'].max() - df['scores'].min()) / (num_thresholds - 1)\n",
    "    threshold = df['scores'].min()\n",
    "    thresholds = [threshold]\n",
    "    while threshold < df['scores'].max():\n",
    "        threshold += threshold_step\n",
    "        thresholds.append(threshold)\n",
    "    return list(reversed(thresholds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need a function to calculate the scores matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def scores(df, coefficients):\n",
    "    '''\n",
    "    helper function to multiply a DataFrame by coefficients\n",
    "    '''\n",
    "    scores = (df * coefficients).sum(axis=1)\n",
    "    scores = pd.DataFrame(scores)\n",
    "    scores.columns = ['scores']\n",
    "    \n",
    "    return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next three functions are necessary to calculate the **Area Under The ROC Curve**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_condition_incidence(df):\n",
    "    total_outcomes = len(df)\n",
    "    total_outcomes_1 = len(df[df['outcomes'] == 1])\n",
    "    total_outcomes_0 = total_outcomes - total_outcomes_1\n",
    "    \n",
    "    condition_incidence = total_outcomes_1 / total_outcomes\n",
    "    \n",
    "    return (total_outcomes, total_outcomes_1, total_outcomes_0, condition_incidence)\n",
    "\n",
    "\n",
    "def true_positive_analysis(df, thresholds):\n",
    "    '''\n",
    "    helper function to calculate the true positives at each threshold\n",
    "    '''\n",
    "    df = df.copy()\n",
    "    for threshold in thresholds:\n",
    "        df[str(threshold)] = df[['scores','outcomes']].apply(lambda x: \n",
    "            x['outcomes'] if x['scores']>threshold else 0, axis=1)\n",
    "\n",
    "    return df\n",
    "    \n",
    "    \n",
    "def false_positive_analysis(df, thresholds):\n",
    "    '''\n",
    "    helper function to calculate the false positives at each threshold\n",
    "    '''\n",
    "    df = df.copy()\n",
    "    for threshold in thresholds:\n",
    "        df[str(threshold)] = df[['scores','outcomes_rev']].apply(lambda x: \n",
    "            x['outcomes_rev'] if x['scores']>threshold else 0, axis=1)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next up is a function to create the **Area Under The Curve matrix**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def area_under_the_curve(df_positive, df_negative, thresholds, total_outcomes, total_outcomes_1, total_outcomes_0):\n",
    "    '''\n",
    "    helper function to calculate the area under the receiver operating \n",
    "    characteristic (ROC) curve and are-under-the-curve\n",
    "    '''\n",
    "    \n",
    "    autc = pd.DataFrame(index=('TPs at threshold','TP rate','FPs at threshold',\n",
    "             'FP rate','Area of Rectangle','FNs at threshold','Cost Per Event'))\n",
    "    tp_rate_prev = 0\n",
    "    fp_rate_prev = 0\n",
    "\n",
    "    for threshold in thresholds:\n",
    "        threshold = str(threshold)\n",
    "        true_positives = df_positive[threshold].sum(axis=0)\n",
    "        autc.set_value('TPs at threshold', threshold, true_positives)\n",
    "        tp_rate = true_positives / total_outcomes_1\n",
    "        autc.set_value('TP rate', threshold, tp_rate)\n",
    "        false_positives = df_negative[threshold].sum(axis=0)\n",
    "        autc.set_value('FPs at threshold', threshold, false_positives)\n",
    "        fp_rate = false_positives / total_outcomes_0\n",
    "        autc.set_value('FP rate', threshold, fp_rate)\n",
    "        if threshold != thresholds[0]:\n",
    "            area_of_rectangle = ((tp_rate + tp_rate_prev) * \n",
    "                                (fp_rate - fp_rate_prev)) / 2\n",
    "        tp_rate_prev = tp_rate\n",
    "        fp_rate_prev = fp_rate\n",
    "        autc.set_value('Area of Rectangle', threshold, area_of_rectangle)\n",
    "        autc.set_value('FNs at threshold', threshold, total_outcomes_1 - true_positives)\n",
    "        total_area = autc.sum(axis=1)['Area of Rectangle']\n",
    "        if total_area < .5:\n",
    "            total_area = 1 - total_area\n",
    "        \n",
    "    return autc, total_area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, a function to calculate the **minimum-cost-per-event**, which is only called if the **cost-per-false-positive** and **cost-per-false-negative** are known."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def minimum_cost_per_event(autc, cost_per_false_positive, cost_per_false_negative):\n",
    "    '''\n",
    "    helper function to calculate the minimum cost per event for a given matrix\n",
    "    '''\n",
    "\n",
    "    total_outcomes = (len(autc.columns))\n",
    "    autc = autc.T\n",
    "    autc['Cost Per Event'] = (cost_per_false_negative * autc['FNs at threshold'] + \n",
    "        cost_per_false_positive * autc['FPs at threshold']) / total_outcomes\n",
    "    min_cost_per_event = autc['Cost Per Event'].min(axis=0)\n",
    "    min_threshold = autc['Cost Per Event'].idxmin(axis=0)\n",
    "    TP_rate = autc.get_value(min_threshold, 'TP rate')\n",
    "    FP_rate = autc.get_value(min_threshold, 'FP rate')\n",
    "    \n",
    "    return min_cost_per_event, min_threshold, autc, TP_rate, FP_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all the necessary helper function defined, and so now we build a function to run the full model. When called, this function takes the specified features and runs logistic regression to calculate correlations between the features. It then uses the previously defined helper functions to run through a binary classification model. If the **cost per false positive** and **cost per false negative** are not known, the best way to evaluate the model's performance is by maximizing the total **area under the curve**. However, if those values are known, then the function will determine the threshold which corresponds to the **minimum cost per event**. In both cases, the function return all values for both the training data and the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def run_model(features, cost_per_false_positive=None, cost_per_false_negative=None):\n",
    "    training = training_raw.copy()\n",
    "    test = test_raw.copy()\n",
    "    \n",
    "    y_values = 'outcomes ~ ' + ' + '.join(features)\n",
    "\n",
    "    features.insert(0, 'intercept')\n",
    "    \n",
    "    y, X = dmatrices(y_values,training,return_type='dataframe')\n",
    "    \n",
    "    model = LogisticRegression(fit_intercept = False).fit(X,y.values.ravel())\n",
    "    \n",
    "    model_scores = scores(training[features],model.coef_[0])\n",
    "\n",
    "    model_scores['outcomes'] = training['outcomes']\n",
    "    model_scores['outcomes_rev'] = training['outcomes_rev']\n",
    "    \n",
    "    total_outcomes, total_outcomes_1, total_outcomes_0, condition_incidence = calc_condition_incidence(training)\n",
    "    \n",
    "    model_thresholds = calc_thresholds(model_scores, 200)\n",
    "    \n",
    "    TP_model_scores = true_positive_analysis(model_scores, model_thresholds)\n",
    "    FP_model_scores = false_positive_analysis(model_scores, model_thresholds)\n",
    "    \n",
    "    autc_matrix, autc = area_under_the_curve(TP_model_scores, FP_model_scores,model_thresholds, \n",
    "                            total_outcomes, total_outcomes_1, total_outcomes_0)\n",
    "    \n",
    "    min_cost_per_event = None\n",
    "    \n",
    "    if cost_per_false_positive != None and cost_per_false_negative != None:\n",
    "        min_cost_per_event, min_threshold, min_cost_matrix, TP_rate, FP_rate = minimum_cost_per_event(autc_matrix, \n",
    "                              cost_per_false_positive, cost_per_false_negative)\n",
    "        \n",
    "        prob_TP = TP_rate * condition_incidence\n",
    "        prob_FP = FP_rate * condition_incidence\n",
    "        classification_incidence = prob_TP + prob_FP\n",
    "    else:\n",
    "        min_cost_per_event = None\n",
    "        min_threshold = None\n",
    "        TP_rate = None\n",
    "        FP_rate = None\n",
    "        prob_TP = None\n",
    "        prob_FP = None\n",
    "        classification_incidence = None\n",
    "    \n",
    "    test_scores = scores(test[features],model.coef_[0])\n",
    "    test_scores['outcomes'] = test['outcomes']\n",
    "    test_scores['outcomes_rev'] = test['outcomes_rev']\n",
    "    test_outcomes, test_outcomes_1, test_outcomes_0, test_condition_incidence = calc_condition_incidence(training)\n",
    "    \n",
    "    TP_test_scores = true_positive_analysis(test_scores, model_thresholds)\n",
    "    FP_test_scores = false_positive_analysis(test_scores, model_thresholds)\n",
    "    test_autc_matrix, test_autc = area_under_the_curve(TP_test_scores, FP_test_scores, model_thresholds, \n",
    "                            test_outcomes, test_outcomes_1, test_outcomes_0)\n",
    "    min_cost_per_event_test = None\n",
    "    if cost_per_false_positive != None and cost_per_false_negative != None:\n",
    "        a,b,cost_per_event_matrix, test_TP_rate, test_FP_rate = minimum_cost_per_event(test_autc_matrix, cost_per_false_positive, cost_per_false_negative)\n",
    "        min_cost_per_event_test = cost_per_event_matrix.get_value(min_threshold, 'Cost Per Event')\n",
    "        test_prob_TP = test_TP_rate * test_condition_incidence\n",
    "        test_prob_FP = test_FP_rate * test_condition_incidence\n",
    "        test_classification_incidence = test_prob_TP + test_prob_FP\n",
    "    else:\n",
    "        cost_per_event_matrix = None\n",
    "        test_TP_rate = None\n",
    "        test_FP_rate = None\n",
    "        min_cost_per_event_test = None\n",
    "        test_prob_TP = None\n",
    "        test_prob_FP = None\n",
    "        test_classification_incidence = None\n",
    "    \n",
    "    return (features, model.coef_[0], autc, test_autc, min_threshold, min_cost_per_event, \n",
    "            min_cost_per_event_test, condition_incidence, test_condition_incidence, \n",
    "            prob_TP, test_prob_TP, classification_incidence, test_classification_incidence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the first model to try, let's use a basic feature list with all features available and in a linear format, then let's call the *run_model* function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features1 = ['age','years_employment','years_at_address','income',\n",
    "             'credit_card_debt','automobile_debt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model1 = run_model(features1, cost_per_false_positive, cost_per_false_negative)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to view the results, it is helpful to define another helper function which will format the information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_results(model):\n",
    "    print ('|=================================================================|')\n",
    "    print ('|                   Feature     Coefficient                       |')\n",
    "    print ('|                   -------     -----------                       |')\n",
    "    for idx in range(len(model[0])):\n",
    "        print ('|', '{:>25}'.format(model[0][idx]), ' : ', '{:<25}'.format(\n",
    "                model[1][idx]), '        |')\n",
    "    print ('|                                                                 |')\n",
    "    print ('|                                   Training       Test           |')\n",
    "    print ('|         Area Under the ROC Curve:', '{:<10}'.format(\n",
    "            round(float(model[2]), 5)), '   ', '{:<10}'.format(\n",
    "                    round(float(model[3]), 5)), '    |')\n",
    "    if model[4] != None:\n",
    "        print ('|           Minimum Cost Per Event:', '{:<10}'.format(\n",
    "                round(float(model[5]), 5)), '   ', '{:<10}'.format\n",
    "            (round(float(model[6]), 5)), '    |')\n",
    "        print ('|                     at threshold:', '{:<20}'.format\n",
    "               (round(float(model[4]), 10)), '         |')\n",
    "        print ('|              Condition Incidence:', '{:<10}'.format(\n",
    "                round(float(model[7]), 5)), '   ', '{:<10}'.format(\n",
    "                        round(float(model[8]), 5)), '    |')\n",
    "        print ('|    Probability of True Positives:', '{:<10}'.format(\n",
    "                round(float(model[9]), 5)), '   ', '{:<10}'.format(\n",
    "                        round(float(model[10]), 5)), '    |')\n",
    "        print ('|  Test (Classification) Indidence:', '{:<10}'.format\n",
    "               (round(float(model[11]), 5)), '   ', '{:<10}'.format(\n",
    "                       round(float(model[12]), 5)), '    |')\n",
    "    print ('|=================================================================|')\n",
    "    print ('')\n",
    "    \n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view the results from this first model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  0.002751246713909548              |\n",
      "|                       age  :  -0.014182070323539155             |\n",
      "|          years_employment  :  -0.23551163634047798              |\n",
      "|          years_at_address  :  0.01238765406651646               |\n",
      "|                    income  :  -1.8770999393600802e-05           |\n",
      "|          credit_card_debt  :  -0.0004000758764461891            |\n",
      "|           automobile_debt  :  -7.202038663058489e-05            |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.839          0.83207        |\n",
      "|           Minimum Cost Per Event: 637.5          700.0          |\n",
      "|                     at threshold: -0.7266747506                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.18           0.17           |\n",
      "|  Test (Classification) Indidence: 0.21833        0.205          |\n",
      "|=================================================================|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_results(model1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the difference **Area Under the ROC Curve** for the training data and the test data, we can tell that this model is quite robust. If we didn't know the **cost per false positive** and **cost per false negative**, we'd stop here and know our model is pretty good. But with that information included, we can see that **Minimum Cost Per Event** for both training and test data are also very close. This also tells us our threshold: -0.7266747506.\n",
    "\n",
    "We now know that for a bank to maximize profits, they should put each application to a test of:\n",
    "\n",
    "      0.002751246713909548\n",
    "    + age * -0.014182070323539155\n",
    "    + years_employment * -0.23551163634047798\n",
    "    + years_at_address * 0.01238765406651646\n",
    "    + income * -1.8770999393600802e-05\n",
    "    + credit_card_debt * -0.000400075876446189\n",
    "    + automobile_debt * -7.202038663058489e-05\n",
    "\n",
    "If this value is greater than the threshold, **-0.7266747506**, the bank should reject the credit card application. In the data given, there is a 25% default rate on 600 applications, with a cost per false negative of \\$5000. If the bank were to simply accept all applications, the defaulted credit card accounts would cost the bank\n",
    "\n",
    "    25% * $5000 * 600\n",
    "    = $750,000\n",
    "    \n",
    "which results in a cost per application event of 750,000 / 600 = **\\$1250**. This logistic regression/binary classification model has a cost per event of only \\$700 on the test set, which indicates that this model would save the bank 1250 - 700 = **$550** per application, ***if we assume that these 600 applications are representative of all future applications***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's compare this model with a few others:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===========Model 1===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  0.002751246713909548              |\n",
      "|                       age  :  -0.014182070323539155             |\n",
      "|          years_employment  :  -0.23551163634047798              |\n",
      "|          years_at_address  :  0.01238765406651646               |\n",
      "|                    income  :  -1.8770999393600802e-05           |\n",
      "|          credit_card_debt  :  -0.0004000758764461891            |\n",
      "|           automobile_debt  :  -7.202038663058489e-05            |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.839          0.83207        |\n",
      "|           Minimum Cost Per Event: 637.5          700.0          |\n",
      "|                     at threshold: -0.7266747506                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.18           0.17           |\n",
      "|  Test (Classification) Indidence: 0.21833        0.205          |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 2===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  -0.46993811557001974              |\n",
      "|                       age  :  -0.014257718093444458             |\n",
      "|          years_employment  :  -0.13975465967856912              |\n",
      "|          years_at_address  :  0.35889294895437956               |\n",
      "|            debt_to_income  :  -2.6330215415207308               |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.79133        0.72613        |\n",
      "|           Minimum Cost Per Event: 771.14428      1019.9005      |\n",
      "|                     at threshold: -0.3338926189                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.125          0.16           |\n",
      "|  Test (Classification) Indidence: 0.145          0.21833        |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 3===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  0.0028124058394982136             |\n",
      "|                       age  :  -0.013827247430510198             |\n",
      "|          years_employment  :  -0.2388204968671745               |\n",
      "|          years_at_address  :  0.012622709943160995              |\n",
      "|                    income  :  -1.8788886758204342e-05           |\n",
      "|          credit_card_debt  :  -0.0004020324151019304            |\n",
      "|           automobile_debt  :  -7.241280640241991e-05            |\n",
      "|            debt_to_income  :  -0.0044535498176817255            |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.8394         0.83227        |\n",
      "|           Minimum Cost Per Event: 634.32836      696.51741      |\n",
      "|                     at threshold: -0.7257425855                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.18           0.17           |\n",
      "|  Test (Classification) Indidence: 0.21833        0.205          |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 4===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  -3.072025478497639e-15            |\n",
      "|                       age  :  -1.1266313715543904e-13           |\n",
      "|                      age2  :  -4.273407914312796e-12            |\n",
      "|          years_employment  :  -3.6119912862144965e-14           |\n",
      "|         years_employment2  :  -5.331617870607429e-13            |\n",
      "|          years_at_address  :  -2.0833435925189577e-15           |\n",
      "|         years_at_address2  :  -2.6848810587327176e-15           |\n",
      "|                    income  :  -1.3191127830827888e-10           |\n",
      "|                   income2  :  -7.182807806041723e-11            |\n",
      "|          credit_card_debt  :  6.0151537231961465e-12            |\n",
      "|         credit_card_debt2  :  1.3736481308216482e-08            |\n",
      "|           automobile_debt  :  1.1601824478614247e-11            |\n",
      "|          automobile_debt2  :  -7.408933426123217e-10            |\n",
      "|            debt_to_income  :  3.3654092921208067e-16            |\n",
      "|           debt_to_income2  :  -7.508443912955794e-18            |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.68727        0.72393        |\n",
      "|           Minimum Cost Per Event: 1019.9005      982.58706      |\n",
      "|                     at threshold: 0.0614954064                  |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.095          0.135          |\n",
      "|  Test (Classification) Indidence: 0.12833        0.17667        |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 5===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  0.23781458628036495               |\n",
      "|                       age  :  -0.07659921615147447              |\n",
      "|                      age2  :  0.00103474079581916               |\n",
      "|          years_employment  :  -0.15472291885712175              |\n",
      "|         years_employment2  :  0.0009679305621018377             |\n",
      "|          years_at_address  :  0.7414429920707974                |\n",
      "|         years_at_address2  :  -0.1603989125227616               |\n",
      "|            debt_to_income  :  -2.293499977351208                |\n",
      "|           debt_to_income2  :  1.3693624843390073                |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.8044         0.72453        |\n",
      "|           Minimum Cost Per Event: 750.0          1012.5         |\n",
      "|                     at threshold: -0.3324833904                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.125          0.145          |\n",
      "|  Test (Classification) Indidence: 0.14167        0.19167        |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 6===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  1.0679706156153228                |\n",
      "|                  age_norm  :  -0.668320402189066                |\n",
      "|     years_employment_norm  :  -2.272657387828949                |\n",
      "|     years_at_address_norm  :  0.686104210923637                 |\n",
      "|               income_norm  :  0.14634392508400001               |\n",
      "|     credit_card_debt_norm  :  -1.2487954353784745               |\n",
      "|      automobile_debt_norm  :  -0.4726352145070143               |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.7894         0.75827        |\n",
      "|           Minimum Cost Per Event: 687.5          912.5          |\n",
      "|                     at threshold: -0.8535983774                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.17           0.135          |\n",
      "|  Test (Classification) Indidence: 0.20833        0.17833        |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 7===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  -0.9553731714784415               |\n",
      "|                  age_norm  :  -0.7153714031236724               |\n",
      "|     years_employment_norm  :  -2.1243680196368064               |\n",
      "|     years_at_address_norm  :  0.5453947169655253                |\n",
      "|            debt_to_income  :  -2.572261311507994                |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.80627        0.75707        |\n",
      "|           Minimum Cost Per Event: 725.0          962.5          |\n",
      "|                     at threshold: -0.6249701785                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.14           0.14           |\n",
      "|  Test (Classification) Indidence: 0.16333        0.18333        |\n",
      "|=================================================================|\n",
      "\n",
      "===========Model 8===========\n",
      "|=================================================================|\n",
      "|                   Feature     Coefficient                       |\n",
      "|                   -------     -----------                       |\n",
      "|                 intercept  :  1.0215884585797725                |\n",
      "|                  age_norm  :  -0.4912250625367655               |\n",
      "|     years_employment_norm  :  -2.0046461910247326               |\n",
      "|     years_at_address_norm  :  0.7387093650189749                |\n",
      "|       debt_to_income_norm  :  -2.268339220485032                |\n",
      "|                                                                 |\n",
      "|                                   Training       Test           |\n",
      "|         Area Under the ROC Curve: 0.80607        0.75973        |\n",
      "|           Minimum Cost Per Event: 696.51741      907.9602       |\n",
      "|                     at threshold: -0.7474566497                 |\n",
      "|              Condition Incidence: 0.25           0.25           |\n",
      "|    Probability of True Positives: 0.145          0.14           |\n",
      "|  Test (Classification) Indidence: 0.16833        0.18           |\n",
      "|=================================================================|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print ('===========Model 1===========')\n",
    "print_results(model1)\n",
    "\n",
    "features2 = ['age','years_employment','years_at_address','debt_to_income']\n",
    "model2 = run_model(features2, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 2===========')\n",
    "print_results(model2)\n",
    "\n",
    "features3 = ['age','years_employment','years_at_address','income',\n",
    "             'credit_card_debt','automobile_debt', 'debt_to_income']\n",
    "model3 = run_model(features3, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 3===========')\n",
    "print_results(model3)\n",
    "\n",
    "features4 = ['age','age2','years_employment','years_employment2',\n",
    "             'years_at_address','years_at_address2','income','income2',\n",
    "             'credit_card_debt','credit_card_debt2','automobile_debt',\n",
    "             'automobile_debt2','debt_to_income','debt_to_income2']\n",
    "model4 = run_model(features4, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 4===========')\n",
    "print_results(model4)\n",
    "\n",
    "features5 = ['age','age2','years_employment','years_employment2',\n",
    "             'years_at_address','years_at_address2','debt_to_income','debt_to_income2']\n",
    "model5 = run_model(features5, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 5===========')\n",
    "print_results(model5)\n",
    "\n",
    "features6 = ['age_norm','years_employment_norm','years_at_address_norm','income_norm',\n",
    "             'credit_card_debt_norm','automobile_debt_norm']\n",
    "model6 = run_model(features6, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 6===========')\n",
    "print_results(model6)\n",
    "\n",
    "features7 = ['age_norm','years_employment_norm','years_at_address_norm','debt_to_income']\n",
    "model7 = run_model(features7, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 7===========')\n",
    "print_results(model7)\n",
    "\n",
    "features8 = ['age_norm','years_employment_norm','years_at_address_norm','debt_to_income_norm']\n",
    "model8 = run_model(features8, cost_per_false_positive, cost_per_false_negative)\n",
    "print ('===========Model 8===========')\n",
    "print_results(model8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the **area under the curve** of each model, and the values of and differences between the **minimum cost per events**, we can see that Model 1, the simpliest model, is actually very nearly the best. The only model to edge it out is Model 3, which adds the additional feature of **debt to income ratio**. Because this model is slightly more complex though and the gain is very minimal, I am tempted to use Model 1. There is a risk of overfitting when model complexity increases so I do not think the minimal gains are worth the additional complexity. Therefore, Model 1 is the bank's best bet.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "*IF*\n",
    "**0.00275 - [age] \\* 0.0142 - [years_employment] \\* 0.236 + [years_at_address] \\* 0.0124 - [income] \\* 0.0000188 - [credit_card_debt] \\* -0.000400 - [automobile_debt] \\* 0.0000720**\n",
    "*is greater than* **-0.7266747506**,\n",
    "***reject*** the applicant"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
